Add Accuracy in time #75
Conversation
There was a problem hiding this comment.
I think the current implementation may be misleading when the time_grid does not have a good resolution (for instance small n_time_grid).
There are two tau values:
- one used in
y_test_class, it's the specified value (byquantilesortaus) and returned value as well - one used in
y_pred_class, it's the least greater value intime_grid, given bytime_grid[tau_idx]
I see two options:
- raise a warning if these two are too "different" (but difficult to tell what too different means as it will depend on the CIF rate of change)
- actually redefine and return
tau = time_grid[tau_idx]used for bothy_pred_classandy_test_class, in other wordstauis forced to be intime_grid.
WDYT ?
| .. [Alberge2024] J. Alberge, V. Maladiere, O. Grisel, J. Abécassis, G. Varoquaux, | ||
| "Survival Models: Proper Scoring Rule and Stochastic Optimization | ||
| with Competing Risks", 2024 |
There was a problem hiding this comment.
I think I saw an earlier remark of @judithabk6 about this, but you should update this to the aistats ref and congrats btw :)
There was a problem hiding this comment.
I added the HAL link, for AISTATS I guess we should wait for the proceeding link? Because the PDF is not available on OpenReview anymore. WDYT?
There was a problem hiding this comment.
I agree that's a good reason to wait
| for tau in taus: | ||
| mask_past_censored = (y_test["event"] == 0) & (y_test["duration"] <= tau) | ||
|
|
||
| tau_idx = np.searchsorted(time_grid, tau) |
There was a problem hiding this comment.
Yes, although that's a common assumption, it might be good to sort it and sort the columns of y_pred? If so, would you raise a warning? Or should we throw an error when time_grid is not sorted? 🤔
There was a problem hiding this comment.
I added a warning, and a reordering with a copy
You're right. To extend your suggestion 2, would you agree with dropping the time_grid = np.array([5, 10])
quantiles = np.array([.0, .2, .5, .8])
taus = np.quantile(time_grid, quantiles)
tau_indices = np.searchsorted(time_grid, taus)
taus = np.unique(time_grid[tau_indices])
taus
>> [5, 10] |
Yes, it seems a good solution to me ! And if I remember correctly taus = np.quantile(time_grid, quantiles, method='inverted_cdf')does just that. |
Address #74
Need to document and add test